Repository navigation
fix(ssh): a relay that reports a PTY absent must authorise a fresh spawn - #16763
Conversation
03d73fb to
5ac2552
Compare
|
Reviewers: the file count below is temporarily wrong. This PR is 6 files, not 58.
The actual change is 6 files: (three modified, two new test files) — commits Holding the rebase until the base settles; the diff will snap back to 6 files then. Verified in advance that the rebase is clean: all three modified files, plus Review P2-1 (persistence coverage for the absence fallback) is addressed in |
bf0b963 to
92de673
Compare
92de673 to
f450562
Compare
reattachSshPtySession flattened the relay's `PTY "<id>" not found` into a bare `SSH_SESSION_EXPIRED`, discarding the evidence. isPtyAlreadyGoneError classifies on that message via /PTY ".+" not found/, so it could never match — and attachStablePaneOwner's existing already-gone fallback (retire the pane binding, onPtyExit, spawn fresh) is gated on exactly that predicate. The recovery machinery was already correct; only the classification was broken, which is why `pty:spawn` rejected outright after a relay restart renumbered its PTYs from pty-1. SshPtyAbsentFromRelayError carries the verdict the message cannot: a reachable relay answered for this exact id and reported it absent. Both thrown messages stay byte-identical to the previous ones, so all six message-based consumers are untouched; only callers that can act on the stronger verdict test the class. Deliberately not raised for a transport failure, timeout, disposed multiplexer, identity mismatch, or restoreRequired — none of those observe the process, and treating them as absence would orphan live remote work.
The first cut passed no store/worktreeId, so attachStablePaneOwner's guard short-circuited and retireTerminalSurfaceFromPersistence — which deletes the parent tab and its layout when the retired leaf is the last one — never ran. retirePersistedStablePaneOwner and attachStablePaneOwner had no coverage at all.
99496a9 to
bd6ff90
Compare
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
📝 WalkthroughWalkthroughThe change adds ✨ Finishing Touches📝 Generate docstrings
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…awn (stablyai#16763) * fix(ssh): a relay that reports a PTY absent must authorise a fresh spawn reattachSshPtySession flattened the relay's `PTY "<id>" not found` into a bare `SSH_SESSION_EXPIRED`, discarding the evidence. isPtyAlreadyGoneError classifies on that message via /PTY ".+" not found/, so it could never match — and attachStablePaneOwner's existing already-gone fallback (retire the pane binding, onPtyExit, spawn fresh) is gated on exactly that predicate. The recovery machinery was already correct; only the classification was broken, which is why `pty:spawn` rejected outright after a relay restart renumbered its PTYs from pty-1. SshPtyAbsentFromRelayError carries the verdict the message cannot: a reachable relay answered for this exact id and reported it absent. Both thrown messages stay byte-identical to the previous ones, so all six message-based consumers are untouched; only callers that can act on the stronger verdict test the class. Deliberately not raised for a transport failure, timeout, disposed multiplexer, identity mismatch, or restoreRequired — none of those observe the process, and treating them as absence would orphan live remote work. * test(ssh): cover the absence fallback against real persistence The first cut passed no store/worktreeId, so attachStablePaneOwner's guard short-circuited and retireTerminalSurfaceFromPersistence — which deletes the parent tab and its layout when the retired leaf is the last one — never ran. retirePersistedStablePaneOwner and attachStablePaneOwner had no coverage at all. (cherry picked from commit 25fa6a9)
…awn (stablyai#16763) * fix(ssh): a relay that reports a PTY absent must authorise a fresh spawn reattachSshPtySession flattened the relay's `PTY "<id>" not found` into a bare `SSH_SESSION_EXPIRED`, discarding the evidence. isPtyAlreadyGoneError classifies on that message via /PTY ".+" not found/, so it could never match — and attachStablePaneOwner's existing already-gone fallback (retire the pane binding, onPtyExit, spawn fresh) is gated on exactly that predicate. The recovery machinery was already correct; only the classification was broken, which is why `pty:spawn` rejected outright after a relay restart renumbered its PTYs from pty-1. SshPtyAbsentFromRelayError carries the verdict the message cannot: a reachable relay answered for this exact id and reported it absent. Both thrown messages stay byte-identical to the previous ones, so all six message-based consumers are untouched; only callers that can act on the stronger verdict test the class. Deliberately not raised for a transport failure, timeout, disposed multiplexer, identity mismatch, or restoreRequired — none of those observe the process, and treating them as absence would orphan live remote work. * test(ssh): cover the absence fallback against real persistence The first cut passed no store/worktreeId, so attachStablePaneOwner's guard short-circuited and retireTerminalSurfaceFromPersistence — which deletes the parent tab and its layout when the retired leaf is the last one — never ran. retirePersistedStablePaneOwner and attachStablePaneOwner had no coverage at all.
7 of 7 in the SSH stack, on top of #16752 (#16746 -> #16747 -> #16748 -> #16750 -> #16751 -> #16752 -> this). The diff below is only this change.
ELI5
If the relay program on the remote machine dies and restarts, it forgets its shells and numbers new ones from 1 again.
So Orca asks for "pty-1", and the fresh relay honestly answers "I don't have a pty-1."
Orca already knows how to handle that — throw away the stale link and start a new shell. But the message got flattened into a generic "session expired" error on the way, so the code that checks for "gone" couldn't recognise it, and the recovery never ran. You got a terminal that refused to open.
The fix: keep the specific answer instead of flattening it, so the recovery that already exists can fire.
Why this is safe: the app only treats a shell as gone when the relay is reachable and says so — and in one case only after the relay has checked the operating system and confirmed the process is dead. A dropped connection or a timeout never counts, because "I couldn't reach it" is not "it's gone." That distinction was tested by running the real check against every error the connection layer can produce.
It can never kill anything — worst case is a shell left running unattached, which was already possible before this change through a different route.
The bug
A relay renumbers its PTYs from
pty-1on every start. After the relay daemon dies and restarts, a pane still bound topty-1takes the attach path against a PTY that no longer exists, andpty:spawnrejects outright:The recovery machinery already existed and was already correct.
attachStablePaneOwner(src/main/ipc/pty/pane/stable-owner.ts:242) has exactly the right fallback — retire the pane binding,runtime.onPtyExit, then letspawnForStablePanespawn fresh — and it is gated onisPtyAlreadyGoneError.The defect is one layer up.
reattachSshPtySession(src/main/providers/ssh-pty-session-reattach.ts:227-231) flattened the relay'sPTY "pty-1" not foundinto a bareError("SSH_SESSION_EXPIRED: pty-1"), discarding the evidence.isPtyAlreadyGoneErrorclassifies on that message via/PTY ".+" not found/, so it could never match, and the error propagated to the renderer instead of falling back. Observed call path:This also explains why an earlier attempt in
kill.tschanged nothing: wrong path entirely.The fix
SshPtyAbsentFromRelayErrorcarries the verdict the message cannot — a reachable relay answered for this exact id and reported it absent. Perdocs/reference/ssh-execution-boundary.mdthat isexited, notunverifiable.isPtyAlreadyGoneErroralso accepts the class.Why this is narrow, not just plausible
Negative space.
pty.attachissrc/relay/pty-handler.ts:1763, and only three throws are inside it::1776:1789isProcessAlive(pid), found it dead, marked the physical exit, notified its exit listener, disposed the managed PTY and removed it from its table — then threw:1801:1789is the strongest form of the argument: the relay is not reporting ignorance, it has verified death and acted on it. "Not found" reads weaker than it is. The other fournot foundthrows (:958,:1942,:2042,:2051) are in different methods and unreachable from this path.Loss of contact cannot be mistaken for absence. Every string
ssh-channel-multiplexer.tscan synthesise, run against the real/PTY ".+" not found/ipredicate::53SSH connection lost, reconnecting...:53Multiplexer disposed:72Request "<m>" timed out after <n>ms:486Method not found: <m>:229,:261Request "<m>" was cancelled:607Connection timed out (no ack received)The sole place an Error is built from a relay-supplied payload is
handleResponse(:520), so only the two genuine relay strings can ever reach the predicate.restoreRequired(the PTY is live, only its source stream is not) keeps its plain Error and is never classified absent.Blast radius is one call site.
isPtyAlreadyGoneError's other consumers —kill.ts:66,kill.ts:266,inspect.ts:91— all handlepty.shutdown/pty.inspectProcesserrors. The new class is thrown only fromreattachSshPtySession, so none of them can ever see it.stable-owner.ts:242is the only behaviour that changes.Both thrown
error.messagevalues are byte-identical to the previous ones. The old code builtSSH_SESSION_EXPIRED: <id>${mismatchMarker}where the marker was' SSH_PTY_IDENTITY_MISMATCH'or''; the two branches produce exactly those strings.The string that crosses IPC does change, and that is the more important half. Electron serializes the stack, whose first line is
${name}: ${message}, so the renderer now seesSshPtyAbsentFromRelayError: SSH_SESSION_EXPIRED: pty-1where it previously sawError: SSH_SESSION_EXPIRED: pty-1. That is visible in this PR's own e2e observable below. Every consumer was therefore checked against the changed string, and none anchors onError:or on a line start — all six match on substring or on a regex with no start anchor. That is why it is safe, not an incidental detail:src/main/ipc/pty/ipc/spawn-execute.ts:145-146src/main/ipc/pty/runtime/spawn-execute.ts:225-226(the paired-runtime path — easy to miss, there are two)TerminalErrorToast.tsx:32, the fussiest consumer:SSH_SESSION_EXPIRED:[ \t]*\S*(?:[ \t]+SSH_PTY_IDENTITY_MISMATCH)?ipc-pty-connect.ts:124,remote-runtime-pty-transport.ts:1526,pty-connection/ssh-session-connect.ts:15This is parity with shipped local behaviour, not a new policy. The local daemon already reaches this exact fallback:
SessionNotFoundErrorcarries the messageSession not found: <id>(daemon-errors.ts:59), whichisPtyAlreadyGoneErroralready matches via/Session not found/i(liveness.ts:65). Local panes have been retiring their binding and respawning on positive absence for as long as that predicate has existed. This PR gives SSH the same treatment for the same class of evidence.Can this abandon or duplicate a LIVE remote session?
It can never kill one — the fallback issues no shutdown or kill, only client-side bookkeeping plus a spawn. The failure mode is orphaning, never termination.
The one real risk, stated plainly: the relay is authoritative only over its own id space. If a daemon is abnormally replaced while the old daemon's PTYs survive, the new daemon truthfully answers "pty-1 not found" while a live shell exists under the old one, and we retire the binding and spawn fresh — orphaning it. This is pre-existing: the renderer already did exactly this on
SSH_SESSION_EXPIRED(deferred-session-reattach-connect.ts:144-150); this PR moves the same decision into main for the stable-pane path. Bounded to one extra shell per pane per relay restart.I previously called this unavoidable. It is not, and the PR should not claim it is. Abnormal replacement is one line away:
src/relay/relay-daemon.ts:23-27handlesuncaughtExceptionwithrelayLogLine+socketOwnership.cleanup()+process.exit(1)and no PTY disposal, so any uncaught throw in the relay orphans every PTY it owned. And the client is not actually blind here —relay.statusalready returnspidanduptimeMs(relay-daemon.ts:130-131), so a daemon demonstrably younger than the binding is detectable. Two non-blocking follow-ups are filed: dispose PTYs in that handler, and decline to re-runlaunchAgent/resumeProviderSessionwhen the answering daemon is younger than the binding.The fresh spawn uses the original request's options, so a request carrying
command/launchAgent/resumeProviderSessionwill run it — pre-existing cold-restore behaviour, not introduced here.Concurrency:
attachStablePaneOwnerre-resolves the owner before and after retiring and aborts withterminal_pane_owner_changedif anything moved;retirePersistedStablePaneOwneris fenced on the exact ptyId + incarnation. It cannot clobber a pane that re-bound underneath it, and it cannot loop.Evidence
The e2e oracle cannot discriminate this change in either direction —
ssh-lost-kill-tab-resurrection.spec.tspasses at the stack tip withSshPtyAbsentFromRelayErrorabsent from the build. Don't go looking for oracle evidence that cannot exist. The real evidence is below.Two grep counts, identical across both runs on this branch, carry the e2e evidence. The pass/fail is incidental.
grep -c "Error occurred in handler for 'pty:spawn'"grep -c "resurrected the closed tab"Primary e2e observable (deterministic, where the test outcome is not): the presence or absence of
Disable the new clause and that line returns; enable it and it is gone, with the run otherwise byte-identical. A/B'd on one tree, one build pipeline.
Unit A/B: with the clause disabled,
stable-pane-relay-absence-respawn.test.tsfails on exactly one case ("spawns fresh once the relay has positively answered for that id") and the others pass. The clause is load-bearing.Persistence coverage (added after review). The first cut of that suite passed no
store/worktreeId, soattachStablePaneOwner's guard (stable-owner.ts:258-261) short-circuited and the retirement never ran — which matters, becauseretireTerminalSurfaceFromPersistencedeletes the parent tab and its layout when the retired leaf is the last one.retirePersistedStablePaneOwnerandattachStablePaneOwnerhad zero coverage repo-wide. Three tests now drive the fallback against a real in-memory session: a two-leaf tab keeps the tab and the sibling's binding; a one-leaf tab drops from persistence (pinning current behaviour, since the fallback returnsowner: nullso tab survival then rests on the renderer); and a non-absence failure leaves persistence byte-for-byte untouched. A/B'd by removingworktreeId— the two persistence tests fail, the rest pass, confirming they exercise the path rather than the guard.Two full oracle runs on this branch, provenance
SshPtyAbsentFromRelayError=6 inout/main/index.jsand tombstones=13 in the renderer each time. Both runs: test 2 passed all three cycles; test 1 failed at the cycle-2createRemoteTerminalTabsetup timeout.grep -c "resurrected the closed tab"= 0 in both — any red here is a setup timeout, never a correctness failure.Test 1's red is a separate, pre-existing transport defect that this PR does not touch and did not introduce (own ticket). Its discriminator, in one line:
It is intermittent, not deterministic: red 5 of 5 in isolated single-spec runs, green in the full 14-spec lane.
Gates:
pnpm tcclean · 314 files / 3849 tests passed, 0 failed (providers, ipc/pty, persistence, ssh, tombstones, toast humanizer) ·check:code-quality:changed0 findings across 55 files · oxlint + oxfmt clean · nomax-linesdisables.